Crate erl_tokenize

Erlang source code tokenizer.

Examples

Tokenizes the Erlang code io:format("Hello").:

use erl_tokenize::Tokenizer;

let src = r#"io:format("Hello")."#;
let tokenizer = Tokenizer::new(src);
let tokens = tokenizer.collect::<Result<Vec<_>, _>>().unwrap();

assert_eq!(tokens.iter().map(|t| t.text()).collect::<Vec<_>>(),
           ["io", ":", "format", "(", r#""Hello""#, ")", "."]);

References

• erl_scan module
• Erlang Data Types

Modules

tokens	Tokens.
values	Token values.

Structs

Error	This crate specific error type.
Lexer	Lexer.
Position	Position of token.
Tokenizer	Tokenizer.

Enums

ErrorKind	The list of the possible error kinds
HiddenToken	Hidden token.
LexicalToken	Lexical token.
Token	Token.

Traits

PositionRange

This trait allows to get a (half-open) range where the subject is located.

Type Definitions

Result

This crate specific Result type.